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Abstract 

Because of the mixed research-based nature of literature reviews, it is surprising, then, that 
insufficient information has been provided as to how reviewers can incorporate mixed 
research approaches into their literature reviews. Thus, in this article, we provide a mixed 
methods research approach—Q methodology—for analyzing information extracted from 
literature reviews. Specifically, after describing the history and characteristic of Q 
methodology, we show how this approach can be mapped onto the literature review 
process—a process that we call a Q Methodology Research Synthesis (QMRS). In particular, 
we outline the steps involved in a QMRS. We contend that our framework represents a first 
step in an attempt to help literature reviewers analyze and interpret information extracted 
from literature reviews in an optimally rigorous way. 

Keywords: Q methodology, Q Methodology Research Synthesis, Literature review, 
Literature review process, Mixed methods research, Mixed research 
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1. Introduction 

Conducting the literature review represents the most difficult component of the research 
process—regardless of the type of empirical research study represented (i.e., qualitative 
research study, quantitative research study, or mixed research study). Indeed, when 
conducting the literature review, researchers (i.e., literature reviewers) face numerous 
challenges. First, as noted by Onwuegbuzie and Frels (2016), when the goal of the literature 
review is to inform primary research, then the literature reviewer should conduct a series of 
literature reviews, as needed, throughout the conduct of the primary research. Specifically, 
the review of the literature can inform any or all of the 12 components of a primary research 
report: problem statement, literature review, theoretical/conceptual framework, research 
question(s), hypotheses, participants, instruments, procedures, analyses, interpretation of the 
findings, directions for future research, and implications for the field (Onwuegbuzie & Frels, 
2016). Simply put, the literature review process does not end at the onset of the primary study. 
That is, the literature review typically should take place throughout the research 
process—that is, before, during, and after the primary research study (Onwuegbuzie & Frels, 
2016). Therefore, with very few exceptions (e.g., grounded theory research; Glaser & Strauss, 
1967), wherein some researchers argue against conducting an initial literature review before 
data collection (for an excellent discussion, see McGhee, Marland, & Atkinson, 2007), the 
literature review can be the most intense and time-consuming component of the research 
process, especially when the extant literature for the underlying topic is extensive. 

A second challenge of the literature review process stems from the fact that it is not a linear 
process (Onwuegbuzie, Collins, Leech, Dellinger, & Jiao, 2010). Although the literature 
review typically precedes the primary research study in most instances, it is very common for 
researchers to oscillate between the primary research study and the extant information. This 
non-linearity adds complexity to the literature review process. 

A third challenge is that literature reviews are not value neutral (Dellinger, 2005; 
Onwuegbuzie & Frels, 2012). Indeed, in every case, literature reviewers make a series of 
decisions such as what sources are used to infonn the literature review, what literature is 
included and excluded, what literature is supported or criticized, and so on. Consequently, 
any given literature review can be framed in numerous ways that reflect the value system of 
the literature reviewer. 

A fourth challenge is that a literature review involves much more than a review of literature, 
unlike its name (i.e., “literature review” or “review of the literature”) falsely suggests 
(Onwuegbuzie, Leech, & Collins, 2011). Indeed, as noted by Onwuegbuzie and Frels (2016), 
in addition to reviewing printed and digital published and unpublished literature, reviewers 
should consider extracting knowledge to inform their literature reviews via such means as 
formally or informally interviewing (e.g., via face-to-face, email, Skype) experts in the topic 
area and reviewing visual data (e.g., drawings, photographs, videos) or collecting data that 
represent digital text (e.g., via Web 2.0 platforms such as Facebook and Twitter). Reviewing 
information from these additional modes also increases the complexity of the literature 
review process. 
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A fifth challenge of the literature review stems from the use of the literature review as a 
methodology (Onwuegbuzie & Frels, 2016) because of its potential to have a “coherent 
foundation for inquiry with tightly interconnected logics of justification, positioning, 
procedures, and rationales” (Greene, 2006, p. 94). Indeed, supporting this contention is the 
fact that, optimally, the literature review process involves the collection, analysis, and 
interpretation of both qualitative and quantitative data (i.e., infonnation)—regardless of 
whether the source of information represents a quantitative, qualitative, or mixed research 
study (Onwuegbuzie et ah, 2010; Sandelowski, Voils, & Barroso, 2006). For instance, at the 
very least, the following elements of each empirical source that informs literature 
reviews—whether representing quantitative, qualitative, or mixed research studies—contain 
quantitative information: 

> Sample size(s) pertaining to every quantitative, qualitative, and mixed research study 
selected for the literature review analysis and synthesis 

> Findings (e.g., descriptive statistics, score reliability, p values, effect sizes, confidence 
interval, meta-analysis information) pertaining to each quantitative research study and mixed 
research study presented in the literature review section of the cited work 

> Findings (e.g., descriptive statistics, score reliability, p values, effect sizes, confidence 
intervals, meta-analysis information) presented in the results section of each quantitative 
research study and mixed research study selected for the literature review. 

Also, the following elements of the research study contain qualitative information: 

> Information about the sample characteristics pertaining to every quantitative, qualitative, 
and mixed research study selected for the literature review analysis and synthesis 

> Findings (e.g., codes, themes, meta-themes, metaphors, quotations, narrative) pertaining 
to each qualitative research study presented in the literature review section of the work cited 

> Findings (e.g., themes, meta-themes, metaphors, quotations, narrative) presented in the 
results section of each qualitative research study or mixed research study selected for the 
literature review. 

> Information from the discussion/conclusion section of every quantitative, qualitative, and 
mixed research study selected for the literature review. 

Because of the array of quantitative and qualitative data that are potentially inherent in each 
work, every literature review lends itself simultaneously to the analysis of quantitative and 
qualitative information. Consequently, every literature review optimally involves using mixed 
research techniques (Onwuegbuzie et ah, 2010; Onwuegbuzie & Frels, 2016). Simply put, 
analyzing and synthesizing both quantitative and qualitative information within the same 
literature review automatically renders the literature review process as a mixed research study 
(Onwuegbuzie et ah, 2010). Indeed, with respect to the analysis of information, optimally, the 
reviewer should be competent in conducting quantitative-based (e.g., meta-analysis; Glass, 
1976), qualitative-based (e.g., meta-synthesis; Sandelowski & Barroso, 2003), and mixed 
research-based (e.g., meta-summary; Sandelowski & Barroso, 2003) analyses. 
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Due to the mixed research-based nature of literature reviews, it is surprising, then, that 
although several authors have attempted to make the literature review process more 
warranted and transparent by providing a step-by-step guide to conducting literature reviews 
(i.e., Bettany-Saltikov, 2012; Combs, Bustamante, & Onwuegbuzie, 2010a, 2010b; Cronin, 
Ryan, & Coughlan, 2008; Dellinger & Leech, 2007; Fink, 2009; Garrard, 2009; Hart, 2005; 
Jesson, Matheson, & Lacey, 2011; Leech, Dellinger, Brannagan, & Tanaka, 2010; Machi & 
McEvoy, 2009; Onwuegbuzie et al, 2010; Onwuegbuzie & Frels, 2012, 2014; Onwuegbuzie, 
Leech, & Collins, 2012; Ridley, 2012), these authors have provided insufficient information 
as to how reviewers can incorporate mixed research approaches into their literature reviews. 
We have identified only seven frameworks, to date, that apply the principles of mixed 
research: (a) Whittemore and Knafl (2005), called integrative review, (b) Gaber (2000), 
called meta-needs assessment, (c) Harden and Thomas (2010), called mixed methods 
synthesis', (d) Sandelowski et al. (2006), called mixed research synthesis', (e) Pluye, Gagnon, 
Griffiths, and Johnson-Lafleur (2009), called mixed studies review, (f) Pawson, Greenhalgh, 
Harvey, and Walshe (2005), called realist review, and, most recently; (g) Onwuegbuzie et al. 
(2010), also called a mixed research synthesis. However, many more works of this type are 
needed. Thus, the purpose of this article is to provide a mixed methods research 
approach—specifically, Q methodology—for analyzing information extracted from literature 
reviews, which yields a process that we call a Q Methodology Research Synthesis (QMRS). 

2. Theoretical Framework 

Q methodology, which was developed in the mid 1930s by William Stephenson after he 
received a Ph.D. in both Physics and Psychology, involves examining correlations among 
participants across a set (i.e., sample) of variables that culminates in a reduction of the 
participants’ many viewpoints to a few factors, which are assumed to represent shared 
feelings, beliefs, opinions, perspectives, or preferences (Newman & Ramlo, 2010) via any of 
the four sources of qualitative data identified by Leech and Onwuegbuzie (2008), namely: 
talk (i.e., data that are extracted directly from the voices of the participants using data 
collection techniques such as individual interviews and focus groups), observations (i.e., 
collection of data by systematically watching or perceiving one or more events, interactions, 
nonverbal communication in order to address or to infonn one or more research questions), 
images (i.e., still [e.g., drawings, photographs] or moving [e.g., videos] visual data that are 
observed or perceived), and documents (i.e., collection of text that exists either in printed or 
digital fonn). 

Because Q methodology involves the use of factor analysis, historically, it has been deemed 
as representing a quantitative research approach. However, because the study of subjectivity 
has been associated more with the qualitative research tradition, and because Q methodology 
typically involves the use of relatively small samples, recently, Q methodology has been 
reframed as representing a mixed methodology (Ernest, 2011; Newman & Ramlo, 2010), that 
involves “a successful combination of the two differing styles of research” (Ray & 
Montgomery, 2006, p. 3). Simply put, the qualitative component of Q methodology provides 
a forum for participants to express their subjective opinions and the quantitative component 
of Q methodology involves the use of factor analytic data reduction and induction to yield 
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insights regarding the formation of perceptions, opinions, and the like, as well as to generate 
testable hypotheses (Valenta & Wigger, 1997). Moreover, Q methodology “provides a way to 
investigate empirically how an individual, separately or as part of a group, thinks about a 
topic or issue of interest” (Durning, 2007, p. 1678) while, at the same time, retaining the 
individual’s point of view (McKeown & Thomas, 1988; Newman & Ramlo, 2010). As noted 
by Valenta and Wigger (1997), Q-methodology research emphasizes the qualitative how and 
why people think the way that they do; the methodology does not count how many people 
think a certain way. The goal of Q-methodology is, first and foremost, to uncover different 
patterns of thought (not their numerical distribution among the larger population) (p. 502). 

3. Mapping Q Methodology onto the Literature Review Process: QMRS 

An important part of the search process stage of the literature review process is for the 
reviewer to identify the experts of the underlying topic. Once these experts have been 
identified, they could be contacted by the reviewer and asked to participate in a Q 
methodology study. The first step of the Q (methodology) study would involve the 
development of a set of, say, 40 items (i.e., statements)—called the concourse —that evolve 
from a thorough analysis of the literature review information on the topic of interest. For 
example, this analysis could represent one or more of the 17 qualitative data analysis 
approaches identified by Onwuegbuzie et al. (2012) for analyzing and interpreting literature. 
Table 1 presents these 17 approaches. Typically, the number of coucourse items ranges from 
30 to 60 items. 


Table 1. Possible qualitative analyses for research syntheses 


Type of Analysis 

Short Description of Analysis 

Constant comparison 
analysis 

Systematically reducing source(s) to codes inductively, then developing 
themes from the codes. These themes may become headings and subheadings 

in the literature review section. 

Classical content analysis 

Systematically reducing source(s) to codes deductively or inductively, then 
counting the number of codes. 

Word count 

Counting the total number of (key)words used or the number of times a 
particular word is used either during a within-study or between-study 
literature analysis. 

Keywords-in-context 

Identifying keywords and utilizing the surrounding words to understand the 
underlying meaning of the keyword in a source or across sources. 

Domain analysis 

Utilizing the relationships between symbols and referents to identify domains 
in a source(s). 

Taxonomic analysis 

Creating a classification system that categorizes the domains in a pictorial 
representation (e.g., flowchart) to help the literature reviewer understand the 
relationships among the domains. 


94 


www.macrothink.org/jei 











,11 Macrothink 
All Institute" 


Journal of Educational Issues 

ISSN 2377-2263 
2015, Vol. 1, No. 2 


Componential analysis 

Using matrices and/or tables to discover the differences among the 
subcomponents of domains. 

Theme analysis 

Involves a search for relationships among domains, as well as a search for 
how these relationships are linked to the overall cultural context. 

Discourse analysis 

Selecting representative or unique segments of language use, such as several 
lines of an interview transcript involving a researcher, and then examining 
the selected lines in detail for rhetorical organization, variability, 
accountability, and positioning. This analysis is particularly useful when 
reviewing literature review sections of empirical articles, literature review 
articles, theoretical/conceptual articles, and methodological articles. 

Secondary data analysis 

Analyzing pre-existing sources or artifacts. 

Membership categorization 
analysis 

Examining how authors/researchers communicate research terms, concepts, 
findings, and categories in their works. 

Semiotics 

Using talk and text as systems of signs under the assumption that no meaning 
can be attached to a single term. This form of analysis shows how signs are 
interrelated for the purpose of creating and excluding specific meanings. 

Manifest content analysis 

Describing observed (i.e., manifest) aspects of communication via objective, 
systematic, and empirical means. 

Qualitative comparative 
analysis 

Systematically analyzing similarities and differences across sources, typically 
being used as a theory-building approach, allowing the reviewer to make 
connections among previously built categories, as well as to test and to 
develop the categories further. This analysis is particularly useful for 
assessing causality in findings across sources. 

Narrative analysis 

Considering the potential of stories to give meaning to research findings, and 
treating data as stories, enabling reviewers to reduce data to a summary. 

Text mining 

Analyzing naturally occurring text within multiple sources in order to 
discover and capture semantic information. 

Micro-interlocutor analysis 

Analyzing information stemming from one or more focus groups of 
researchers, scholars, or practitioners about which participant(s) responds to 
each question, the order that each participant responds, the characteristics of 
the response, the nonverbal communication used, and the like. 


Note. Adapted from “Qualitative analysis techniques for the review of the literature,” by A. J. 
Onwuegbuzie, N. L. Leech, K. M. T. Collins, 2012, The Qualitative Report, 17 (Art . 56), p. 
12. Copyright 2012 by Anthony J. Onwuegbuzie, Nancy L. Leech, Kathleen M. T. Collins 
and Nova Southeastern University. 
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Now, let us assume that the reviewer developed 40 statements, which would represent what is 
called the Q sample. The goal of the Q sample is to provide a microcosm of the larger 
phenomenon of interest—that is, to provide statements that are representative of diverse 
opinions around a topic of interest. Then, the reviewer randomly assigns each of the 40 
statements a number from 1 to 40. The Q participants (i.e., the selected experts) then are 
asked to sort (i.e., subjectively) each statement in comparison to the other statements along a 
continuum anchored by conceptually opposite ratings such as “most agree” to “most disagree,” 
“most like me” to “most unlike me,” or “most like my view” to “least like my view.” During 
this process of sorting, the reviewer places these 40 statements into a distribution that 
approximates the normal distribution that is represented by a grid (e.g., sorting the statements 
along a 9-point quasi-normal distribution from -4 to +4; cf. Figure 1). The Q participants then 
sort these 40 statements either face-to-face (if convenient for them) or via an online sorting 
format that involves the use of the free FlashQ program (www.hackert.biz/flashq). 


Most Most 

Disagree Agree 



Figure 1. Example of a Q sort grid allowing participants to sort 30 statements into nine 
categories that range from “Most Disagree” to “Most Agree” that yield a quasi-normal forced 

distribution 


At this stage of the Q methodology process, qualitative techniques (e.g., face-to-face/virtual 
interviews, face-to-face/virtual focus groups) can be used to understand the participants’ 
rationales for sorting the statements in order to facilitate the quality of inferences that the 
reviewer can derive from the Q methodology study. Once the statements are sorted by the 
participants, the reviewer then subjects the Q sorts to an exploratory-based factor analysis 
that involves factor extraction and factor rotation, which lead to the identification (i.e., 
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flagging) of the experts who are represented by each factor, and which yield factor analyses 
and descriptions for each factor that only involve the experts who are flagged on that factor. 

The reviewer can create one or more of the following four types of tables associated with Q 
methodology: (a) factor scores, (b) rank-ordered list of Q items (i.e., statements) alongside z 
scores to create a representative sort for each emergent factor, (c) the list of statements that 
distinguish each factor from other factors, and (d) the list of consensus statements that depict 
agreement among all the factors (Newman & Ramlo, 2010). Conveniently, the reviewer can 
use software programs that have been developed specifically to facilitate the Q sort analysis 
(e.g., PQ Method; Schmolck, 2002). These software programs allow the integration of 
quantitative and qualitative data (Newman & Ramlo, 2010). Thus, Q methodology involves 
conducting a mixed analysis (i.e., a mixing or combining of quantitative and qualitative 
analyses; Onwuegbuzie & Combs, 2010) to interpret the quantitative data generated by the 
factor analysis informed by qualitative data of the interrelationships among the statements, 
involving the search for themes (i.e., factors), with the goal of systematically identifying 
categories, connecting them, and searching for discontinuing evidence (Ernest, 2011). 

Q methodology involves both quantitizing (e.g., converting statements to a quasi-normal 
distribution that subsequently is factor analyzed; Miles & Huberman, 1994; Onwuegbuzie & 
Teddlie, 2003; Sandelowski, Voils, & Knafl, 2009; Tashakkori & Teddlie, 1998) and 
quantizing (e.g., forming narrative profiles (Tashakkori & Teddlie, 1998)) for each emergent 
factor) within the same analysis. Interestingly, Q methodology can be transformed to a mixed 
analysis to an even greater degree by conducting follow-up quantitative analyses (e.g., 
correlating the experts’ views with other variables of interest (e.g., demographic variables)) 
and qualitative analyses (e.g., conducting and analyzing follow-up interviews/focus groups to 
confirm or to disconfirm inferences that emerge from the factor analysis). Thus, Q 
methodology represents an extremely powerful methodology for conducting a comprehensive 
and rigorous literature review via the use of experts. 

4. Heuristic Example of a QMRS 

As an illustration, we discuss our ongoing study of how mixed methods research, or what we 
call mixed research (cf. Johnson, Onwuegbuzie, & Turner, 2007), is defined in the literature. 
In what follows, we outline a seven-step process that consist of the four steps that have been 
undertaken and the three planned future steps that will comprise our Q methodology. 

5. Completed Q Methodology Steps 

Step 1. Our first step was to conduct a comprehensive review of the literature (e.g., articles, 
book chapters, books, dissertations and theses, monographs, encyclopedias, government 
documents, trade catalogues, legal and public records information) to identify the various 
ways that authors are defining mixed methods research. This search led us not only to locate 
an array of definitions but also to identify the authors whose definitions were being cited the 
most. 

Step 2. The second step was to interview several of these well-cited authors to find out what 
their latest definitions of mixed methods research was. This notion of interviewing prolific 
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authors-researchers related to the underlying topic (i.e., experts ) has been conceptualized by 
Onwuegbuzie and Frels (2016) as being one of the five broad ways of expanding a literature 
review search and ensuring that a comprehensive literature review takes place. As part of a 
mixed research study to document the challenges faced by instructors and students in mixed 
research courses (cf. Frels, Onwuegbuzie, Leech, & Collins, 2012), 11 leading mixed 
methodologists (i.e., experts) were asked about their preferred mixed methods research 
definition. Interestingly, a significant proportion of these mixed methodologists revealed that 
they utilize some or all of the definition developed by Johnson et al. (2007). 

Johnson et al. (2007) asked 31 leading mixed methodologists to share their definitions of 
mixed methods research. A total of 19 of these mixed methodologists provided updated 
definitions. Using constant comparison analysis (Glaser & Strauss, 1967) to analyze these 19 
definitions, Johnson et al. (2007) extracted themes, which subsequently they used to 
conceptualize an inclusive definition (i.e., consensus) of mixed research. 

Because Johnson et al.’s (2007) article has been the most read and most cited article in the 
area of mixed research since it was published in 2007, with more than 2,000 citations at the 
time of writing this article, we decided to subject their definition, as well as those definitions 
provided by the 19 participants in their study, to a Q methodology. These definitions 
provided the concourse, or initial set of statements. 

Step 3. Unitizing the aforementioned definitions appearing in Johnson et al.’s (2007) article 
yielded 30 statements that ensured comprehensiveness and representativeness with regard to 
the concourse. These 30 statements formed the Q sample. The content-related validity of the 
Q sample was maximized by leaving the statements in each mixed methodologist’s own 
words, edited only slightly for grammar and readability. 

Step 4. Each of these 30 statements was numbered randomly and printed on a separate index 
card to form a set of statements. This procedure was repeated multiple times in order to 
obtain several sets of statements such that each set was identical. Next, we identified the 
sample of N members whom we will ask to participate in the Q Sort phase of our study. 
Currently, we are seeking Institutional Review Board (IRB) approval to conduct the Q Sort 
phase. This participant sample is referred to as person samples or P-sets. The sample size will 
be sufficiently large “to establish the existence of a factor for purposes of comparing one 
factor with another” (Brown, 1980, p. 192)—specifically, that produce at last two or three Q 
sorts that are statistically significant via a factor analysis (Watts & Stenner, 2005). 

6. Future Q Methodology Steps 

Step 5. Once we receive IRB approval, our next step is to conduct the Q Sort phase. This 
phase will involve the participants expressing their subjectivity by modeling their viewpoints 
via the Q Sort. Specifically, these participants will be asked to ra nk order the Q sample 
stimuli (i.e., the mixed methods research definitions) according to what is called a condition 
of interaction, which will involve them sorting the Q statements (i.e., 30 statements) into nine 
categories that range from “Most Disagree” to “Most Agree”—which represent what is called 
a forced-rank continuum (see Figure 1)—in this case, from -4 (i.e., statements with which the 
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participants most disagree) to 4 (i.e., statements with which the participants most agree), with 
a neutral viewpoint being represented by 0. The Q sorts either will occur face-to-face or 
online via the free FlashQ program (www.hackert.biz/flashq). This ranking of each of the 30 
statement by the participants within a fixed distribution will result in them rating the relative 
value of each statement with respect to their personal viewpoints. The participants who 
conduct the Q sort under both conditions (i.e., face-to-face and online) also will be asked to 
type their rationales for their statement choices. For the majority of participants who are 
expected to conduct their Q sorts online, once completed, the FlashQ webhosting site will 
automatically email us their arrangements, alongside their rationale(s) for statement choices 
and demographic infonnation. 

Step 6. Once all arrangements from the Q sorts have been obtained, the analysis phase will 
begin. This analysis phase will involve intercorrelating the NQ (i.e. N x 30) sorts as variables 
and factor-analyzing the N x N correlation matrix, which “allows those of similar views to be 
grouped into factors” (Newman & Ramlo, 2010, p. 508). In other words, the factor analysis 
will identify patterns of viewpoints that emerge within and across the participants (McKeown 
& Thomas, 1988). PQMethod 2.11 (Schmolck, 2002), a free computer software program, will 
be used to conduct this analysis. Specifically, a principal component analysis will be used to 
conduct the analysis wherein the total variance of each statement will be used to assess the 
shared variation among the statements. Additionally, a varimax (i.e., an orthogonal) rotation 
will be employed. Correlations will be considered statistically significant at the 1% level if 
they are above ±2.58 times the standard error, and statistically significant at the 5% level if 
they are above ±1.96 times the standard error. 

The factors (i.e., patterns) that will emerge are called subjective operants (McKeown & 
Thomas, 1988), and the participant’s association with these subjective operants will be 
characterized by the magnitude of the “pattern coefficients” and “structure coefficients.” 
Factor scores then will be computed for each factor, which will yield a parsimonious set of 
“composite Q sorts” called factor arrays (McKeown & Thomas, 1988, p. 6) that capture, via 
qualitative and quantitative analyses, the different viewpoints that are contained within the 
larger concourse (Ernest, 2011; McKeown & Thomas, 1988; Newman & Ramlo, 2010). Each 
factor array will represent a generalization of a subjective viewpoint regarding the definition 
of mixed methods research. The factor scores provide a quantitative way “to assess the 
significance of different statement locations within different factor arrays” (McKeown & 
Thomas, 1988, p. 6). 

Prior to merging the Q sorts in order to generate the model factor array, factor weights will be 
calculated using the generated structure/pattem coefficients, on the basis that some sorts are 
closer statistically to one factor than another and, therefore, will receive a higher score. As 
recommended by McKeown and Thomas (1988), these scores initially will be calculated as z 
scores and subsequently converted to whole numbers (±4 to -4) to aid in the comparison of 
the factor array comparisons. As such, it is the weighted statements that will be used to 
interpret the factors. 

As part of the analysis, we will arrive at a list of statements that distinguish each factor from 
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other factors. In additions, we will create a table of consensus statements that depict 
agreement among all the factors (Newman & Ramlo, 2010). In Q methodology, consensus 
statements are “statements that are similar across the various factors based upon statistical 
analyses” (Ramlo, 2011, p. 33). Our consensus table will reveal where consensus exists 
among pairs of factors (i.e., viewpoints). 

Step 7. In addition to conducting the Q sorts, each participant will be asked to take part in an 
interview (i.e., post-sort interview) so that they could explain their sorting arrangement 
choices, especially the highest and lowest rankings in their Q sorts. These interviews will 
occur face-to-face or virtually (e.g., Skype, GoToMeeting) such that both verbal and 
nonverbal data can be collected (cf. Denham & Onwuegbuzie, 2013; Onwuegbuzie & Byers, 
2014). Further, a second post-sort interview will be conducted on key informants, who are 
represented by participants whose structure/pattern coefficients are the highest and, thus, will 
be considered as being most representative of each extracted factor. The purpose of this 
second interview will be twofold: (a) to member check their responses (Manning, 1997) and 
(b) to obtain a richer understanding of their statement arrangement choices (Brown, 1980; 
McKeown & Thomas, 1988). These post-sort interviews will allow for the participants’ 
voices to be heard (i.e., via quotations) regarding their sorting choices, thereby providing 
greater richness to the analysis. 

Interpretive phenomenological analysis (IPA) will be used to analyze responses from the 
post-sort interviews (Smith, 1996a, 1996b). This analysis particularly is suited to Q 
methodology because it is rooted in the theoretical lens of phenomenology and interactionism. 
Along the phenomenological axis, IPA focuses on idiographic (i.e., understanding of unique, 
subjective phenomenon) analysis of meanings, experiences, and subjectivity (Onwuegbuzie 
& Denham, 2014). Specifically, in this phase of our Q methodology, we will use IPA as a 
double-hermeneutic process in which we are attempting to make sense of the P-set 
participants as they attempt to make sense of what mixed methods research means to them, 
with our sense-making process being augmented by detailed and systematic examination of 
each key informant. Our analysis, which we plan to be iterative, fluid, engaged, and 
multi-directional (Smith, 1996a, 1996b), will involve immersive and intense reading and 
re-reading of all interview transcripts; initial noting on the exploratory level of processes, 
opinions, values, and principles; free textual analysis of exploratory noting (i.e., how is mixed 
methods research defined by the participant?) through descriptive, linguistic, and conceptual 
comments; deconstruction (e.g., de-contextualizing); developing emergent themes; 
identifying connections among themes; abstraction; subsumption; polarization; 
contextualization; numeration; and function (Onwuegbuzie & Denham, 2014). Further, we 
plan to use the following prompts suggested by Larkin, Watts, and Clifton (2006): (a) IPA’s 
phenomenological component: mapping out each participant’s concerns from her/his current 
positioning in the field of mixed methods research, (i.e., How does this person understand the 
mixed methods research field)', (b) IPA’s interpretative component: contextualizing the claims 
made by each participant (i.e., What does this mixed methods research mean for this person?), 
and (c) the overall outcome will be an integrated insight into mixed methods research from 
the participants’ perspectives. 
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We believe that the findings from our Q methodology will provide a much-needed 
understanding of mixed methodologists’ subjective perspectives on mixed methods research. 
In turn, this understanding will help to inform this and future literature reviews that we 
conduct on mixed methods research. 

7. Conclusions 

In this article, we contended that there is scant guidance on how to analyze sources that 
inform a literature review. Thus, we have provided a framework for analyzing and 
interpreting sources that stem from documents and conversations/interviews with key 
researchers, scholars, and/or practitioners—namely, using the QMRS. 

As can be seen from our heuristic example, the power of Q methodology is not that it 
provides findings that can be generalized to the population from which the participants were 
selected (i.e., make external statistical generalizations). Rather, the power of Q methodology 
stems from its ability to lead to analytic findings, wherein the literature reviewer is “striving 
to generalize a particular set of [case study] results to some broader theory”; Yin, 2009, p. 43) 
and that are “applied to wider theory on the basis of how selected cases ‘fit’ with general 
constructs”; Curtis, Gesler, Smith, & Washburn, 2000, p. 1002)—or what Thomas and Baas 
(1992/1993) refer to as “substantive inference ‘about’ a phenomenon” (p. 22). It is the 
participants’ viewpoints that can be generalized according to the types of persons who share 
similar points of view on the topic, under the assumption that the statements that comprise the 
Q sample are representative of the universe of viewpoints on the topic (Brown, 1980; 
McKeown & Thomas, 1988). Alternatively stated, although the findings from a Q 
methodology of information extracted from literature reviews generalize only to those who 
participate in the study, because the Q sample reflects a representative sample of statements 
that are drawn from the concourse, which, in turn, evolves from the comprehensive literature 
review, the factor themes potentially can be generalized to the extant body of research on the 
underlying topic. As such, QMRS represents a useful tool that helps both beginning 
researchers and experienced researchers map the mixed analysis process onto the literature 
review process, thereby yielding a more rigorous review of the literature. 
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Appendix 

Q Sample of 30 Statements Extracted from a Literature Review of How Mixed Methods 
Research is Defined. 


Burke Johnson and Anthony Onwuegbuzie: 

1. Mixed methods research is the class of research where the researcher mixes or combines 
quantitative and qualitative research techniques, methods, approaches, concepts or language 
into a single study or set of related studies. 

2. Mixed methods research is the research paradigm that partners with the philosophy of 
pragmatism in one of its forms (left, right, middle). 

3. Mixed methods research follows the logic of mixed methods research (including the 
logic of the fundamental principle and any other useful logics imported from qualitative or 
quantitative research that are helpful for producing defensible and usable research findings). 

4. Mixed methods research relies on qualitative and quantitative viewpoints, data collection, 
analysis, and inference techniques combined according to the logic of mixed methods 
research to address one’s research question(s). 

5. Mixed methods research is cognizant, appreciative, and inclusive of local and broader 
sociopolitical realities, resources, and needs. 

6. The mixed methods research paradigm offers an important approach for generating 
important research questions and providing warranted answers to those questions. 

7. Mixed methods research should be used when the nexus of contingencies in a situation, 
in relation to one’s research question(s), suggests that mixed methods research is likely to 
provide superior research findings and outcomes. 


Udo Kelle: 

8. Mixed methods means the combination of different qualitative and quantitative methods 
of data collection and data analysis in one empirical research project. 

9. This combination can serve for two different purposes: it can help to discover and to 
handle threats for validity arising from the use of qualitative or quantitative research by 
applying methods from the alternative methodological tradition and can thus ensure good 
scientific practice by enhancing the validity of methods and research findings. Or it can be 
used to gain a fuller picture and deeper understanding of the investigated phenomenon by 
relating complementary findings to each other which result from the use of methods from the 
different methodological traditions of qualitative and quantitative research. 
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Donna Mertens: 

10. Mixed methods research, when undertaken from a transformative stance, is the use of 
qualitative and quantitative methods that allow for the collection of data about historical and 
contextual factors, with special emphasis on issues of power that can influence the 
achievement of social justice and avoidance of oppression. 


Steven Miller: 

11. Mixed methods is a form of evolving methodological inquiry, primarily directed to the 
human sciences, which attempts to combine in some logical order the differing techniques 
and procedures of quantitative, qualitative and historical approaches. 

12. Mixed methods must devote itself to resolving an epistemological set of issues, 
ontological, called the “epistemological link,” which represents the rules and rationales that 
“permit” researchers to proceed mixed methodologically. 

13. Mixed methods must devote itself to resolving an ontological set of issues, adhering to 
some form of “minimal realist” ontology, wherein either social reality is “One” but can be 
accessed by different methods separately or working in conjunction, or social reality is 
multiple in nature and can ONLY be accessed through mixed methods. 

14. Present day attempts to couch mixed methods within some broad notion of pragmatism 
are not satisfactory. 


Janice Morse: 

15. A mixed method design is a plan for a scientifically rigorous research process comprised 
of a qualitative or quantitative core component that directs the theoretical drive, with 
qualitative or quantitative supplementary component(s). These components of the research fit 
together to enhance description, understanding and can either be conducted simultaneously or 
sequentially. 


Isadore Newman: 

16. Mixed methods research as a set of procedures that should be used when integrating 
qualitative and quantitative procedures reflects the research question(s) better than each can 
independently. 

17. The combining of quantitative and qualitative methods should better inform the 
researcher and the effectiveness of mixed methods should be evaluated based upon how the 
approach enables the investigator to answer the research question(s) embedded in the 
purpose(s) (why the study is being conducted or is needed; the justification) of the study. 
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Michael Q. Patton: 

18. Mixed methods represents inquiring into a question using different data sources and 
design elements in such a way as to bring different perspectives to bear in the inquiry and 
therefore support triangulation of the findings. 

19. Using different methods to examine different questions in the same overall study is not 
mixed methods. 


Hallie Preskill: 

20. Mixed methods research refers to the use of data collection methods that collect both 
quantitative and qualitative data. 

21. Mixed methods research acknowledges that all methods have inherent biases and 
weaknesses; and that using a mixed methods approach increases the likelihood that the sum 
of the data collected will be richer, more meaningful, and ultimately more useful in 
answering the research questions. 


Margarete Sandelowski: 

22. I think of mixed methods in terms of either a single primary research study or as a 
program of research. 

23. Mixed methods research is more the use of different methodological approaches together 
in a single study or single program of research. 

24. Mixed methods research can be defined at the technique level as the combination of, e.g., 
purposeful and probability sampling, open-ended and closed-ended data collection techniques, 
and narrative and mutivariable analyses—i.e., in which anything can be 

used together (linked or assimilated into each other) 

25. Mixed methods research can be defined at a larger theoretical/paradigmatic level as using 
divergent approaches to inquiry together. 

26. Mixed methods research does not constitute any combination of two or more things, 
because any research involves the use of two or more of something and the use of experiment 
and survey is two things, but they are informed by one mind (typically 
positivist/objectivist/realist). 


Lyn Shulha: 

27. Methods can be “mixed” in a variety of ways. Sometimes, one method serves another in 
validating and explicating findings that emerge from a dominant approach. On other 
occasions, different methods are used for different parts of the issues being investigated, and 
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in an independent way. In more complex cases, the methods and perspectives are deliberately 
mixed from the beginning of the process. 

28. The resulting interaction of problem, method, and results produce a more comprehensive, 
internally consistent, and ultimately, more valid general approach. 

29. What sets the most complex forms of collaborative mixed method research apart from 
other forms of inquiry is that findings depend as much on the researchers’ capacities to leam 
through joint effort and to construct joint meaning as on their expertise in conventional data 
collection and analysis techniques. 


Abbas Tashakkori and Charles Teddlie: 

30. Mixed methods research is a type of research design in which QUAL and QUAN 
approaches are used in type of questions, research methods, data collection and analysis 
procedures, or in inferences. 


Copyright Disclaimer 

Copyright for this article is retained by the author(s), with first publication rights granted to 
the journal. 

This is an open-access article distributed under the terms and conditions of the Creative 
Commons Attribution license (http://creativecommons.Org/licenses/by/3.0/). 


109 


www.macrothink.org/jei 



